Method and System for Determining Projections in Non-Central Catadioptric Optical Systems

ABSTRACT

A three-dimensional (3D) location of a reflection point of a ray between a point in a scene (PS) and a center of projection (COP) of a camera of a catadioptric system is determined. The catadioptric system is non-central and includes the camera and a reflector, wherein a surface of the reflector is a quadric surface rotationally symmetric around an axis of symmetry. The 3D location of the reflection point is determined based on a law of reflection, an equation of the reflector, and an equation describing a reflection plane defined by the COP, the PS, and a point of intersection of a normal to the reflector at the reflection point with the axis of symmetry.

FIELD OF THE INVENTION

This invention relates generally to catadioptric optical systems, andmore particularly to determining a projection in non-centralcatadioptric optical systems.

BACKGROUND OF THE INVENTION

A catadioptric optical system combines refraction and reflectionprinciples, usually via lenses (dioptrics) and curved mirrors(catoptrics). The catadioptric system includes camera imaging reflectorsor refractors, enabling wide field-of-view (FOV) imaging. Thecatadioptric systems have been used in a wide range of applications,including panoramic imaging and visualization, wide-anglereconstruction, surveillance, and mobile robot and car navigation. Thecatadioptric system can be central or non-central.

Central Catadioptric System

Central catadioptric systems use a camera-mirror pair arranged to enablean effective single viewpoint, i.e., all rays of light forming an imageacquired by the camera sensor intersect at one point. Examples of thecentral catadioptric systems include a perspective camera placed on oneof the foci of a hyperbolic or elliptical mirror, and an orthographiccamera placed on an axis of a parabolic mirror.

Non-Central Catadioptric System

Non-central catadioptric systems are widely used in computer visionapplications. Examples of non-central catadioptric systems include aperspective camera placed outside of a spherical mirror, andconfigurations wherein the camera is not placed on the foci of ahyperbolic or elliptical mirror. In contrast with the centralcatadioptric systems, in the non-central catadioptric systems, the raysdo not intersect at one point. Instead, the rays may intersect along aline, or the rays may be tangent to a circle or to a more complex shape.

In a number of applications, it is important to model non-centralcatadioptric systems, which, in turn, requires determining a projectionof a point in a scene (PS) to a center of projection (COP) of the cameraof the catadioptric system. The non-central catadioptric system does nothave an effective center of projection. The COP refers to the center ofprojection of the physical perspective camera used in a catadioptricsystem.

The projection maps the three-dimensional (3D) PS to a two-dimensional(2D) pixel on an image plane of the camera of the catadioptric system.

For example, if the catadioptric system includes a reflector, such as amirror, the projection of the PS onto an image plane of the camerarequires computing a path of the ray of light from the PS to the COP viamirror reflection. Thus, the point of reflection on a surface of themirror needs to be determined. Similarly, if the catadioptric systemincludes the refractor, e.g., a refractive sphere, two points ofrefraction need to be determined to model the projection.

Analytical solutions of projections for central catadioptric systems areknown. However, there is no analytical solution of projection forgeneral non-central catadioptric systems, when a camera is placed at anarbitrary position with respect to a reflector or a refractor.

Several conventional methods approximate non-central catadioptricsystems as the central catadioptric system that enables analyticalsolution for projection. However, those methods lead to inaccuraciessuch as skewed 3D estimation.

Alternative methods use iterative non-linear optimization byinitializing the point of reflection or refraction using the centralapproximation. However, those methods are time-consuming andinappropriate initialization leads to incorrect solutions. Yet anothermethod uses a general linear camera representation for locallyapproximating a non-central catadioptric camera with an affine modelthat allows analytical projection, but this method also introducesapproximation.

Accordingly, it is desired to provide an analytical solution of theprojection for non-central catadioptric systems and to determineanalytically a three-dimensional (3D) location of at least onereflection point of a ray from the PS to the COP of non-centralcatadioptric systems.

SUMMARY OF THE INVENTION

It is an object of an invention to provide a method for determininganalytically a projection for non-central catadioptric systems.

It is further an object of the invention to provide such a method thatdetermines a reflection point of a ray between a point in a scene (PS)and a center of projection (COP) of a camera of a catadioptric system.

It is further an object of the invention to provide such a method thatdetermines the reflection point of the catadioptric system, wherein theCOP is arranged outside of the axis of symmetry of the reflector.

Some embodiments of invention are based on a realization that areflection plane defined by the COP, the PS, and a point of intersectionof a normal of a reflection point with the axis of symmetry of thereflector can be used to derive an analytical equation, which, inconjunction with a law of reflection and an equation of the reflector,can be used to determined the location of the reflection point.

One embodiment is based on another realization, that if a reflector ofthe catadioptric systems is rotationally symmetric, entire coordinatesystem can be rotated to position the COP on at least one axis. Thisrealization enables the embodiment to reduce a degree of a forwardprojection equation (FPE) in one unknown, wherein the unknown is acoordinate of the reflection point. For example, one embodimentdetermines and solves the FPE of eights degrees.

Accordingly, one embodiment of the invention discloses a method fordetermining a three-dimensional (3D) location of a reflection point of aray between a point in a scene (PS) and a center of projection (COP) ofa camera of a catadioptric system, wherein the catadioptric system isnon-central and includes the camera and a reflector, wherein a surfaceof the reflector is a quadric surface rotationally symmetric around anaxis of symmetry, wherein the reflection point is a point of reflectionof the ray on the surface, and wherein a configuration of thecatadioptric system includes the PS and the COP identified by 3Dlocations. The method includes determining a first equation describing areflection plane defined by the COP, the PS, and a point of intersectionof a normal of a reflection point with the axis of symmetry; determininga second equation based on a law of reflection at the reflection point;determining a third equation as an equation of the reflector; andsolving the first, the second, and the third equations to determine 3Dlocation of the reflection point, wherein the steps of the method areperformed by a processor.

The method may optionally include one or more of the following. Forexample, the configuration of the catadioptric system may be acquired ina first coordinate system, and the method may include rotating the firstcoordinate system around the axis of symmetry to a second coordinatesystem, such that at least one coordinate of the COP is on an axis inthe second coordinate system; and mapping the 3D location of thereflection point to the first coordinate system.

For example, one variation may include constructing, based on the first,the second, and the third equations, a forward projection equation (FPE)of eight degrees in one unknown, wherein the unknown is a firstcoordinate of the 3D location of the reflection point in the secondcoordinate system; solving the FPE to determine the first coordinate;determining a second coordinate and a third coordinate of the 3Dlocation of the reflection point in the second coordinate system basedon the first coordinate and the first, the second, and the thirdequations to produce the 3D location of the reflection point in thesecond coordinate system; and mapping the 3D location of the reflectionpoint to the first coordinate system.

Various embodiments, using the 3D location of the reflection point, maydetermined a pose of the reflector with respect to the camera using abundle adjustment, a sparse 3D point in a scene and/or a dense depth mapof a scene.

Another embodiment discloses a method for determining athree-dimensional (3D) location of a reflection point of a ray between apoint in a scene (PS) and a center of projection (COP) of a camera of acatadioptric system, wherein the catadioptric system is non-central andincludes the camera and a mirror, wherein a surface of the mirror is aquadric surface rotationally symmetric around an axis of symmetry,wherein the reflection point is a point of reflection of the ray on thesurface and partitions the ray into an incoming ray and a reflected ray,and wherein the PS and the COP are identified by 3D locations. Themethod includes acquiring a configuration of the catadioptric system ina first coordinate system of an x axis, a y axis, and a z axis, whereinand the mirror is rotationally symmetric around the z axis, determininga rotation matrix; rotating the first coordinate system to a secondcoordinate system, such that an x coordinate of the COP in the secondcoordinate system is zero; determining, in the second coordinate system,a forward projection equation (FPE) of eight degrees in one unknown,wherein the unknown is a coordinate of the reflection point, and whereinthe determining the FPE is based on an equation of the mirror, anequation describing a reflection plane defined by the COP, the PS, and apoint of intersection of the normal with the axis of symmetry, and anequation describing a law of reflection; determining the 3D location ofthe reflection point in the second coordinate system based on the FPE;and mapping the 3D location of the reflection point to the firstcoordinate system using an inverse of the rotation matrix.

Yet another embodiment discloses a catadioptric system configured fordetermining a three-dimensional (3D) location of a reflection point of aray between a point in a scene (PS) and a center of projection (COP) ofthe catadioptric system, wherein the PS and the COP are identified by 3Dlocations, and the catadioptric system is non-central, comprising: areflector, wherein the reflection point is caused by a reflection of theray on a surface of the reflector, wherein the surface is a quadricsurface rotationally symmetric around an axis of symmetry; a cameraarranged at a distance from the surface, wherein the COP of thecatadioptric system is a COP of the camera; and a processor fordetermining the 3D location of the reflection point based on a law ofreflection, an equation of the reflector, and an equation describing areflection plane defined by the COP, the PS, and a point of intersectionof a normal to the reflector at the reflection point with the axis ofsymmetry.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic of a catadioptric system according to embodimentsof an invention;

FIG. 2 is a block diagram of a method for determining a reflection pointaccording to some embodiments of the invention;

FIG. 3 is a schematic of a rotation of a coordinate system of thecatadioptric system according to some embodiments of the invention; and

FIG. 4 is a block diagram of a method for determining a reflection pointaccording to one embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Method and System Overview

Embodiments of invention are based on a realization that athree-dimensional (3D) location of a reflection point of a ray between apoint in a scene (PS) and a center of projection (COP) of a camera of acatadioptric system can be determined analytically using mapping a 3Dstructure of the catadioptric system on a two-dimensional (2D) planedefined by the COP, the PS, and a point of intersection of a normal of areflection point with an axis of symmetry of a reflector of thecatadioptric system.

Such configurations include non-central catadioptric systems having acamera arranged at a distance from a surface of the reflector, whereinthe surface is quadric and rotationally symmetric around an axis ofsymmetry. Importantly, for some embodiments, there is no restriction ona position of a center of projection (COP) of the camera used in thecatadioptric system. For example, the COP can be arranged outside of theaxis of symmetry.

FIG. 1 shows an example of a configuration of the catadioptric system100. The catadioptric system is non-central, and the surface of thereflector is a quadric surface rotationally symmetric around an axis ofsymmetry, e.g., the axis 110. The COP 120 is positioned on or off theaxis of symmetry. In one embodiment, the camera is a pinhole camera.

Some embodiments determine a three-dimensional (3D) location of onereflection point 160 of a ray between a PS, e.g., the PS 130, and a COP,e.g., the COP 120, of the catadioptric system having a camera, e.g., thecamera 121, arranged at a distance from a surface, e.g., the surface 105of the optical folding element, wherein the reflection point is a pointof reflection of the ray from the surface, such that the folding pointpartitions the ray into an incoming ray 131 and a reflected ray 132. ThePS and the COP are identified by 3D locations.

Some embodiments of the invention determine a reflection plane 140defined by the COP 120, the PS 130, and a point of intersection 150 of anormal 135 of a reflection point 160 with the axis of symmetry z 110.

For the purpose of this description, the z axis of the 3D coordinatesystem is the axis of symmetry, e.g., the axis of the mirror. In oneembodiment, the reflector is a spherical mirror, the axis of symmetrypasses through a center of the spherical mirror and the equation of thereflector is an equation of the spherical mirror.

The configuration of the catadioptric system can be acquired in a worldcoordinate system have an x axis, a y axis, and a z axis, wherein theCOP is COP=[c_(x),c_(y),c_(z)]^(T), wherein T is a transpose operator,wherein the PS is PS=[X; Y,Z]^(T). The incoming ray v_(i) is denoted asv_(i)=M−COP, wherein M denotes the reflection point, a normal nrepresents the normal to the mirror at the reflection point M, and thereflected ray v_(r) is denoted as v_(r)=P−M. The quadric, rotationallysymmetric mirror can be described according to

x ² +y ² +Az ² +Bz−C=0,   (1)

wherein A, B, C are parameters for the surface of the mirror.

From the law of reflection, the following constraints can be derived:

-   -   (a) planarity: the rays v_(i), v_(r), and the normal n lie on        the same plane; and    -   (b) angle constraint: an angle between the incoming ray v_(i)        and the normal n is equal to an angle between the reflected ray        v_(r) and the normal n.

These two constraints are represented in the following reflectionequation

v _(r) =v _(i)−2n(v _(i) ^(T) n)/(n ^(T) n).   (2)

Also, the reflection point M is on the surface of the mirror, and hence,the coordinates of the reflection point satisfy the mirror Equation (1).

FIG. 2 shows a block diagram of a method according embodiments of theinvention. The method determines a three-dimensional (3D) location 199of at least one reflection point of a ray between a point in a scene(PS) and a center of projection (COP) of a camera of the catadioptricsystem 100, according to an acquired 170 configuration 175 of thecatadioptric system. In various embodiments, the configuration 175 isconfiguration of existing catadioptric system or a model of thecatadioptric system. The steps of the method are performed by aprocessor 101.

The reflection plane is described with a first equation 186. A secondequation and a third equation 185 are defined using, respectively, a lawof reflection at the reflection point and an equation of the reflector.The first equation, the second equation, and the third equation are usedto determine and solve 190 a forward projection equation (FPE) in oneunknown, wherein the unknown is a first coordinate 191. Next, the 3Dlocation 199, including the first coordinate, a second coordinate, and athird coordinate, can be determined 195 based on the first coordinateand the first, the second, and the third equations.

One embodiment of the invention is based on another realization that ifthe configuration of the catadioptric system is acquired in a firstcoordinate system, the first coordinate system can be rotated 176 to asecond coordinate system, such that at least one coordinate of the COPis on an axis in the second coordinate system. In this embodiment, thefirst, the second, and the third equations are defined and solved in thesecond coordinate system, such that the 3D location of the reflectionpoint is determined in the second coordinate system, and then mappedback to the first coordinate system.

FIG. 3 shows a schematic of the rotation 176, provided for illustrationpurpose only. Because the mirror is rotationally symmetric around the zaxis, the entire coordinate system can be rotated around the z axis toposition the COP on the y axis, as shown in FIG. 3 (right).

Some embodiments determine a rotation matrix R according to

${R = \begin{bmatrix}{c_{y}/t} & {{- c_{x}}/t} & 0 \\{c_{x}/t} & {c_{y}/t} & 0 \\0 & 0 & 1\end{bmatrix}},$

wherein t=√{square root over (c_(x) ²+c_(y) ²)}, and rotate the firstcoordinate system to a second coordinate system, such that, e.g., an xcoordinate of the COP in the second coordinate system is zero. The newcoordinates of the COP 320 is given by

COP_(R) =R*COP=[0,d _(y) ,d _(z)]^(T),

wherein an operator * is a multiplication.

By setting the x coordinate of the COP to zero, the degree of subsequentequations, e.g. the FPE equation, are reduced. Importantly, the rotation176 is independent of the PS. The new coordinates of the PS 330 afterthe rotation is given by PS_(R)=R*PS=[u,v,w]^(T). Some embodimentsdetermine the 3D location of the reflection point M_(R) 360 in thesecond coordinate system using the COP_(R) 320 and the point ofintersection 350 in the second coordinate system, and then map the 3Dlocation of the reflection point to the first coordinate system using aninverse of the rotation matrix R⁻¹.

FIG. 4 shows a flow chart of a method according to one embodiment. Astep 410 includes rotating the first coordinate system in a secondcoordinate system, such that at least one coordinate of the COP is on anaxis in the second coordinate system. A step 420 includes determining,based on the first, the second, and the third equations, a forwardprojection equation (FPE) of eight degrees in one unknown, wherein theunknown is a first coordinate of the 3D location of the reflection pointin the second coordinate system and solving the FPE to determine thefirst coordinate.

A step 430 includes determining a second coordinate and a thirdcoordinate of the 3D location of the reflection point in the secondcoordinate system based on the first coordinate and the first, thesecond, and the third equations to produce the 3D location of thereflection point in the second coordinate system. Finally, a step 440includes mapping the 3D location of the reflection point in the secondcoordinate system to the first coordinate system.

Determining First Equation Based on Reflection Plane

The reflection point in the second coordinate system isM_(R)=[x,y,z]^(T). The reflection plane π includes the rays v_(i),v_(r), and the normal n. The normal at the reflection point M_(R) can bedetermined from the mirror equation as

n=[x, y, Az+B/2]^(T).   (3)

Because the mirror is rotationally symmetric around, e.g., the z axis,the normal intersects the z axis at a point K=[0,0,z−Az−B/2]^(T), whichalso lies on the reflection plane. Thus, the equation of the reflectionplane can be determined using the points K 350, COP_(R) 320, and PS_(R)330, resulting in

c ₁(z)x+c ₂(z)y+c ₃(z)=0,   (4)

wherein

c ₁(z)=(B+2Az)(d _(y) −v)+2d _(y)(w−z)+2v(z−d ₂)

c ₂(z)=u(B+2d _(z)−2z+2Az)

c ₃(z)=ud _(y)(B+2Az).

The Equation (4) is linear for coordinates x and y. Using this Equation(4), the coordinate x can be defined in terms of the coordinates y and zaccording to

$\begin{matrix}{x = {\frac{{{- {c_{2}(z)}}y} - {c_{3}(z)}}{c_{1}(z)}.}} & (5)\end{matrix}$

Substituting the coordinate x in the mirror equation produces the firstequation IE_(i) according to

IE ₁: (c ₁ ²(z)+c ₂ ²(z))y ²+2c ₂(z)c ₃(z)y+c ₃ ²(z)+c ₁ ²(z)(Az ²+Bz−C)=0.   (6)

The first equation is quadratic in the coordinate y, although thecoefficients are functions of the coordinate z. This equation describesa curve Γ 310, given by the intersection of the reflection plane withthe mirror. The reflection point on this curve has to satisfy the angleconstraint. As described below, the second equation IE₂ determined basedon the law of reflection equation is also quadratic in the coordinate y.By eliminating y between the first IE₁ and the second IE₂ equations, asingle 8^(th) degree equation in z, i.e., the FPE, can be determined.

Determining Second Equation Based on Law of Reflection

The reflected ray v_(r) passes through the point in the scene, PS_(R),yielding,

v _(r)×(PS _(R) −M _(R))=0   (7)

where an operator × is a cross product. The incoming ray v_(i) is

v _(i) =M _(R)−COP_(R) =[x,y−d _(y) ,z−d _(x)]^(T).   (8)

The reflected ray v^(r) can also be determined by substituting v_(i) andn in the reflection equation (2). Then, a substitution of the reflectedray v_(r) and values of PS_(R) and M_(R) in Equation (7) producesfollowing three equations (since the cross product is in 3D)

E ₁ :k ₁₁(z)x+k ₁₂(z)y+k ₁₃(z)xy+k ₁₄(z)y ² +k ₁₅(z)=0.

E ₂ :k ₂₁(z)x+k ₂₂(z)y+k ₂₃(z)xy+k ₂₄(z)=0.

E ₃ :k ₃₁(z)y ² +k ₃₂(z)y+k ₃₃(z)=0.   (9)

The polynomials k₃₁, k₃₂, k₃₃ are determined according to

     k₃₁ = −4 d_(y)(B + 2 w − 2 z + 2 Az)k₃₂ = 4 Cd_(z) + 4 Cw − 8 Cz − B²d_(z) − B²w + 4 Bz² − 2 B²z + 4 BC − 4 A²d_(z)z² − 4 A²wz² + 8 ACz + 4 Bd_(y)v − 4 Bd_(z)w − 4 ABz² + 4 Ad_(z)z² + 4 Awz² − 4 ABd_(z)z − 4 ABwz + 8 Ad_(y)vz − 8 Ad_(z)wzk₃₃ = 4A²vz³ − 4 A²d_(y)z³ − 4 BCv − 4 Cd_(z)v + 4 Cd_(y)w − 4Cd_(y)z + 4 Cvz + B²d_(z)v + B²d_(y)w + 4 Ad_(y)z³ + 4 Bd_(y)z² − B²d_(y)z − 4 Avz³ − 4 Bvz² + 3 B²vz − 4 ABd_(y)z² + 8 ABvz² + 4 Ad_(z)vz² − 4 Ad_(y)wz² + 4 A²d_(z)vz² + 4 A²d_(y)wz² − 8 ACvz + 4 Bd_(z)vz − 4 Bd_(y)wz + 4 ABd_(z)vz + 4 ABd_(y)wz.

The three Equations (9) are not independent, thus any one of them can beselected as the second equation. For example, one embodiment selects theequation E₃, because the equation E₃ is independent of x.

Forward Projection Equation

The forward projection equation (FPE) can be determined, e.g., byeliminate y from the first and the second equations to produce a singleequation in only one unknown, e.g., (z). For example, one embodimentrewrites the first and the second equations as

IE ₁ :k ₄₁(z)y ² +k ₄₂(z)y+k ₄₃(z)=0,

IE ₂ :k ₃₁(z)y ² +k ₃₂(z)y+k ₃₃(z)=0,   (10)

where k₄₁(z)=c₁ ²(z)+c₂ ²(z), k₄₂(z)=2c₂(z)c₃(z) and k₄₃(z)=c₃ ²(z)+c₁²(z)(Az²+Bz−C).

Eliminating y² produces

k₄₁(z)(k₄₃(z)k₃₂²(z) − k₄₂(z)k₃₂(z)k₃₃(z) + k₄₁(z)k₃₃²(z)) − k₃₁(z)(−k₃₃(z)k₄₂²(z) + k₄₃(z)k₃₂(z)k₄₂(z) + 2 k₄₁(z)k₄₃(z)k₃₃(z)) + k₄₃²(z)k₃₁²(z) = 0,      wherein     k₄₁(z) = c₁²(z) + c₂²(z), k₄₂(z) = 2 c₂(z)c₃(z), and     k₄₃(z) = c₃²(z) + c₁²(z)(Az² + Bz − C) = 0  are  polynomials  in  z.

Next, a substitution of y into the first and the second equationsproduces the FPE according to

$\begin{matrix}{y = {- {\frac{{{k_{41}(z)}{k_{33}(z)}} - {{k_{31}(z)}{k_{43}(z)}}}{{{k_{41}(z)}{k_{32}(z)}} - {{k_{31}(z)}k_{42}}}.}}} & (11)\end{matrix}$

According to this embodiment, the FPE is a 8^(th) degree polynomialequation in z. The coefficients in the FPE depend on the known mirrorparameters (A,B,C), the known location of COP_(R) (0,d_(y),d_(z)) andthe known location of PS_(R) (u,v,w). If the rotation 176 is notperformed, the FPE is a 12^(th) degree polynomial equation.

Table 1 lists the degree of FPE for various mirror shapes and cameraplacement. A sign “-” indicates that the central configuration may beunpractical to achieve with that particular mirror and a camera placedon the axis of symmetry of the mirror.

TABLE 1 Camera Placement Off-Axis On-Axis Mirror Non- Non- ShapeParameters Central Central Central General A, B, C 8 6 — Spherical A =1, B = 0, C > 0 4 — Elliptical A > 0, B = 0, C > 0 8 6 2 Hyperbolic A <0, B = 0, C < 0 8 6 2 Parabolic A = 0, C = 0 7 5 2 Conical A < 0, B = 0,C = 0 4 2 — Cylindrical A = 0, B = 0, C > 0 4 2 —

For a general quadric and off-axis camera placement, the FPE has adegree of eight. For a parabolic mirror (A=0, C=0), the degree reducesto seven. For a cylindrical mirror polished on outside (A=0, B=0, C>0),the degree reduces to four for the off-axis camera placement. On-axiscamera configuration with cylindrical mirror is practical when themirror is polished on the inside, which reduces the degree of FPEequation to two. When the parameter A≠0, then the parameter B can be setto zero by shifting the mirror along the z axis. Table 1 also shows thedegree of the FPE for conical (C=0, A<0), hyperbolic (A<0, C<0), andellipsoidal (A>0, C>0) mirrors.

Applications of the Forward Projection Equation

Some embodiments use the forward projection equation for concurrentdetermining 3D points in the scene and parameters of the catadioptricsystem using a bundle adjustment algorithm. Determining 3D points in thescene corresponds to sparse 3D reconstruction, while determining theparameters of the catadioptric system provide corresponds to calibrationof the system.

In one embodiment, the catadioptric system includes a single perspectivecamera imaging multiple parabolic mirrors. The internal parameters ofthe camera are computed separately off-line and the parameters of theshape of the mirrors are known. The bundle adjustment can be preformedby minimizing an error starting from an initial solution. The initialposes of the mirrors with respect to the camera can be obtained based onthe boundaries of the mirrors in the acquired images. The initiallocations of 3D points can be obtained as the center of the shortesttransversal of the respective back-projection rays corresponding to themultiple mirrors. Forward projection is required to compute the error asthe sum of reprojection errors in each iteration in the bundleadjustment. Hence, the analytical forward projection equationsignificantly reduces the processing time of the bundle adjustmentcompared to existing approaches using iterative forward projection.

Some embodiments consider that corresponding image points are estimatedusing a feature matching algorithm, such as scale-invariant featuretransform (SIFT), and invariably contain outliers and false matches.Those embodiments perform the simultaneous sparse 3D reconstruction andcalibration by iterating the bundle adjustment with outlier removal.After each bundle adjustment step, the 3D points whose reprojectionerror is greater than twice the average reprojection error are removed.Repeating the outlier removal multiple times can be efficiently donebecause of the forward projection equation.

Another embodiment uses the forward projection equation forreconstructing a dense depth map of a scene. Dense depth maps can bereconstructed by projecting dense 3D locations in the scene to an imagevia multiple mirrors, and computing a color consistency cost, e.g., sumof the squared difference of colors, among corresponding image pixels.The forward projection equation enables the dense depth reconstructionprocess to be performed efficiently.

EFFECT OF THE INVENTION

The embodiments of the invention advance the field of catadioptricimaging both theoretically and practically. Embodiments provideanalytical equations of forward projections for a broad class ofnon-central catadioptric systems. Some embodiments of the inventionenable an analytical determination of the reflection point in thecatadioptric system, wherein the camera is positioned outside of an axisof symmetry of the mirror.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications can be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as come within the truespirit and scope of the invention.

1. A method for determining a three-dimensional (3D) location of a reflection point of a ray between a point in a scene (PS) and a center of projection (COP) of a camera of a catadioptric system, wherein the catadioptric system is non-central and includes the camera and a reflector, wherein a surface of the reflector is a quadric surface rotationally symmetric around an axis of symmetry, wherein the reflection point is a point of reflection of the ray on the surface, and wherein a configuration of the catadioptric system includes the PS and the COP identified by 3D locations, comprising the steps of: determining a first equation describing a reflection plane defined by the COP, the PS, and a point of intersection of a normal of a reflection point with the axis of symmetry; determining a second equation based on a law of reflection at the reflection point; determining a third equation as an equation of the reflector; and solving the first, the second, and the third equations to determine 3D location of the reflection point, wherein the steps of the method are performed by a processor.
 2. A method of claim 1, wherein the 3D location of the reflection point includes a first coordinate, a second coordinate, and a third coordinate, further comprising: constructing a forward projection equation (FPE) in one unknown from the first, the second, and the third equations, wherein the unknown is the first coordinate; solving the FPE to determine the first coordinate; and determining the second coordinate and the third coordinate based on the first coordinate and the first, the second, and the third equations.
 3. The method of claim 2, wherein the FPE is an equation of eight degrees.
 4. The method of claim 1, wherein the reflector is a spherical mirror, the axis of symmetry passes through a center of the spherical mirror and the equation of the reflector is an equation of the spherical mirror.
 5. The method of claim 1, wherein the configuration of the catadioptric system is acquired in a first coordinate system, further comprising: rotating the first coordinate system around the axis of symmetry to a second coordinate system, such that at least one coordinate of the COP is on an axis in the second coordinate system.
 6. The method of claim 5, wherein the first, the second, and the third equations are defined and solved in the second coordinate system, such that the 3D location of the reflection point is determined in the second coordinate system, further comprising: mapping the 3D location of the reflection point to the first coordinate system.
 7. The method of claim 5, further comprising: constructing, based on the first, the second, and the third equations, a forward projection equation (FPE) of eight degrees in one unknown, wherein the unknown is a first coordinate of the 3D location of the reflection point in the second coordinate system; solving the FPE to determine the first coordinate; determining a second coordinate and a third coordinate of the 3D location of the reflection point in the second coordinate system based on the first coordinate and the first, the second, and the third equations to produce the 3D location of the reflection point in the second coordinate system; and mapping the 3D location of the reflection point to the first coordinate system.
 8. The method of claim 1, further comprising: determining, using the 3D location of the reflection point, a pose of the reflector with respect to the camera using a bundle adjustment.
 9. The method of claim 1, further comprising: determining, using the 3D location of the reflection point, a sparse 3D point in a scene.
 10. The method of claim 1, further comprising: determining, using the 3D location of the reflection point, a dense depth map of a scene.
 11. A method for determining a three-dimensional (3D) location of a reflection point M of a ray between a point in a scene (PS) and a center of projection (COP) of a camera of a catadioptric system, wherein the catadioptric system is non-central and includes the camera and a mirror, wherein a surface of the mirror is a quadric surface rotationally symmetric around an axis of symmetry, wherein the reflection point is a point of reflection of the ray on the surface and partitions the ray into an incoming ray and a reflected ray, and wherein the PS and the COP are identified by 3D locations, comprising the steps of: acquiring a configuration of the catadioptric system in a first coordinate system of an x axis, a y axis, and a z axis, wherein and the mirror is rotationally symmetric around the z axis, the COP is COP=[c_(x),c_(y),c_(x)]^(T), wherein T is a transpose operator, wherein the PS is PS=[X,Y,Z]^(T), wherein the incoming ray v_(i) is v_(i)=M−COP, a normal n is the normal to the mirror at the reflection point M, and the reflected ray v_(r) is v_(r)=P−M; determining a rotation matrix R according to ${R = \begin{bmatrix} {c_{y}/t} & {{- c_{x}}/t} & 0 \\ {c_{x}/t} & {c_{y}/t} & 0 \\ 0 & 0 & 1 \end{bmatrix}},$ wherein t=√{square root over (c_(x) ²+c_(y) ²)}; rotating the first coordinate system to a second coordinate system, such that an x coordinate of the COP in the second coordinate system is zero according to R*COP=[0,d _(y) ,d _(z)]^(T), wherein an operator * is a multiplication; determining, in the second coordinate system, a forward projection equation (FPE) of eight degrees in one unknown, wherein the unknown is a coordinate of the reflection point, and wherein the determining the FPE is based on an equation of the mirror, an equation describing a reflection plane defined by the COP, the PS, and a point of intersection of the normal n with the axis of symmetry, and an equation describing a law of reflection; determining the 3D location of the reflection point in the second coordinate system based on the FPE; and mapping the 3D location of the reflection point to the first coordinate system using an inverse of the rotation matrix R⁻¹.
 12. The method of claim 11, further comprising: determining a third equation as an equation of the mirror according to x ² +y ² +Az ² +Bz−C=0, wherein A, B, and C are parameters of the surface of the mirror; determining a first equation describing the reflection plane according to (c ₁ ²(z)+c ₂ ²(z))y ²+2c ₂(z)c ₃(z)y+c ₃ ²(z)+c ₁ ²(z)(Az ² +Bz−C)=0′ wherein z and y are coordinates of the reflection point, c ₁(z)=(B+2Az)(d _(y) −v)+2d _(y)(w−z)+2v(z−d ₂), c ₂(z)=u(B+2d _(z)−2z+2Az), c ₃(z)=ud _(y)(B+2Az). wherein u, v, and w are coordinates of the PS in the second coordinate system, [u, v, w]^(T)=R*PS; determining a second equation based on law of reflection according to k ₃₁(z)y ² +k ₃₂(z)y+k ₃₃(z)=0, wherein polynomials k₃₁, k₃₂, k₃₃ determined according to      k₃₁ = −4 d_(y)(B + 2 w − 2 z + 2 Az) k₃₂ = 4 Cd_(z) + 4 Cw − 8 Cz − B²d_(z) − B²w + 4 Bz² − 2 B²z + 4 BC − 4 A²d_(z)z² − 4 A²wz² + 8 ACz + 4 Bd_(y)v − 4 Bd_(z)w − 4 ABz² + 4 Ad_(z)z² + 4 Awz² − 4 ABd_(z)z − 4 ABwz + 8 Ad_(y)vz − 8 Ad_(z)wz k₃₃ = 4A²vz³ − 4 A²d_(y)z³ − 4 BCv − 4 Cd_(z)v + 4 Cd_(y)w − 4Cd_(y)z + 4 Cvz + B²d_(z)v + B²d_(y)w + 4 Ad_(y)z³ + 4 Bd_(y)z² − B²d_(y)z − 4 Avz³ − 4 Bvz² + 3 B²vz − 4 ABd_(y)z² + 8 ABvz² + 4 Ad_(z)vz² − 4 Ad_(y)wz² + 4 A²d_(z)vz² + 4 A²d_(y)wz² − 8 ACvz + 4 Bd_(z)vz − 4 Bd_(y)wz + 4 ABd_(z)vz + 4 ABd_(y)wz; and determining the FPE based on the first equation, the second equation, and the third equation.
 13. The method of claim 12, further comprising: determining the FPE according to k₄₁(z)(k₄₃(z)k₃₂²(z) − k₄₂(z)k₃₂(z)k₃₃(z) + k₄₁(z)k₃₃²(z)) − k₃₁(z)(−k₃₃(z)k₄₂²(z) + k₄₃(z)k₃₂(z)k₄₂(z) + 2k₄₁(z)k₄₃(z)k₃₃(z)) + k₄₃²(z)k₃₁²(z) = 0,   wherein  k₄₁(z) = c₁²(z) + c₂²(z),   k₄₂(z) = 2c₂(z)c₃(z), and   k₄₃(z) = c₃²(z) + c₁²(z)(Az² + Bz − C) = 0  are  polynomials  in  z.
 14. The method of claim 11, wherein a shape of the mirror is selected from a group consisting of: a spherical, an elliptical, a hyperbolic, a parabolic, a conical, and a cylindrical shape.
 15. A catadioptric system configured for determining a three-dimensional (3D) location of a reflection point of a ray between a point in a scene (PS) and a center of projection (COP) of the catadioptric system, wherein the PS and the COP are identified by 3D locations, and the catadioptric system is non-central, comprising: a reflector, wherein the reflection point is caused by a reflection of the ray on a surface of the reflector, wherein the surface is a quadric surface rotationally symmetric around an axis of symmetry; a camera arranged at a distance from the surface, wherein the COP of the catadioptric system is a COP of the camera; and a processor for determining the 3D location of the reflection point based on a law of reflection, an equation of the reflector, and an equation describing a reflection plane defined by the COP, the PS, and a point of intersection of a normal to the reflector at the reflection point with the axis of symmetry.
 16. The catadioptric system of claim 15, wherein the reflector is a spherical mirror, the axis of symmetry passes through a center of the spherical mirror and the equation of the reflector is an equation of the spherical mirror.
 17. The method of claim 1, wherein the processor rotates a coordinate system of the catadioptric system around the axis of symmetry to position at least one coordinate of the COP on an axis of the coordinate system and inverses the rotation after the 3D location of the reflection point is determined. 